IMAGE INTERPOLATION 



FIELD OF THE INVENTION 
[01] This invention relates to the techniques of interpolation of images using at 
least two original images. Especially, the invention concerns motion compensa- 
tion and finding desired color information of the picture. 

BACKGROUND OF THE INVENTION 
[02] FIG. 1 represents a simplified example of a known interpolation technique. 
Original images, IMG. 1 and IMG. 2, are used when interpolating a new image 5 
between them. Motion vectors 6 are formed from one of the original pictures to 
the other one. The interpolating image 5 is formed using the motion vectors by in- 
terpolating pixel values F of the interpolating image from the values of the motion 
vectors on the original pictures. 

[03] However, if there exists a moving object in the original pictures, there are 
no companion pixels for all the pixels of one of the original images in the other 
original image. In FIG. 1 an object 1 is moving right. When it is moving, new pix- 
els 7 are coming to the front from the background in IMG. 2. On the other hand, 
the moving object covers the pixels in IMG.2, which would have the companion 
pixels 8 in IMG. 1. Since all the pixels do not have companions in the original im- 
ages, some pixels S in the interpolating image must be tracked by forming mo- 
tion vectors from one of the original images to the interpolating image. As shown 
in FIG. 1, part of the pixels 4 in image 5 are formed using the motion vectors be- 
tween the original images, the other part of the pixels 3 are formed using the mo- 
tion vectors between the first original image IMG. 1 and the interpolating image 5, 
and further the other part of the pixels 2 are formed using the motion vectors be- 
tween the second original image IMG. 2 and the interpolating image 5. 

[04] The pixel values in both the original images contain color information. In 
FIG. 1, object 1 has a certain color C1 in the first image and another color C2 in 



the second image. The object 9, which is formed in the interpolated image 5, has 
a color that is a combination of the colors of the object in the original images. 

[05] The problem with the most commonly known solutions is that forming an 
intermediate image, the motion vectors must be fixed to the intermediate image: 
in other words, each intermediate picture needs motion vectors, which are spe- 
cific for it. Another problem is that the interpolation process is linear, and due to 
this the formed image is linear interpolation from the original images. However, 
the motion information between the original images may contain nonlinear parts, 
which cannot be tracked by using the known interpolation techniques. For ex- 
ample, the motion of objects between two pictures may be such that it is impos- 
sible to track naturally using the known interpolation techniques. The goal of this 
invention is to eliminate the drawback concerning non-linearity of the known so- 
lutions - preferably using a solution, which alleviate the first problem mentioned 
above. 

SUMMARY OF THE INVENTION 
[06] The invention is based primarily on two ideas, or insights. The first insight, 
which is well established in the art, and supports efficient use of the invention is 
that only one vector field of the motion vectors between original pictures is re- 
quired for forming any intermediate picture. The first insight is supportive of the 
preferable implementation of the invention, but however the invention works with 
any vector field. The second insight is that when the desired intermediate picture, 
or a target image, is known beforehand, it is possible to adjust parameter values 
related to the forming of the intermediate picture. By adjusting the parameter val- 
ues, nonlinear motion and color information can be taken into account, and an 
image closer to the target image may thus be formed. 

[07] The first parameter value that is to be adjusted describes the timeline posi- 
tion of the intermediate picture (more precisely the timeline position of a pixel or a 
group of pixels of the intermediate picture). (Pictures, such as video frames are 



normally situated on a time axis in presentation order). This parameter is called 
the shape parameter, because it affects mostly the shape of the objects in the in- 
termediate image - in other words motion between the original pictures is com- 
pensated for. The second parameter describes the source of the color informa- 
tion used when creating the intermediate frame; in practice, both original frames 
are used as the source of the color information, so the second parameter is the 
weight used in calculating the color of the intermediate frame to be created 

[08] When adjusting the shape and color parameters -the parameter value pair - 
related to the whole image or to a restricted area of the image, a search of the 
best parameter values can be done simultaneously for both shape and color us- 
ing e.g. a logarithmic search algorithm. Furthermore, for adjusting the parameter 
values additional adjusting parameters can be used. The invention makes it pos- 
sible to encode, for example, videos in a very efficient and accurate way. 

[09] An inventive method comprises the fundamental steps of: 

a) forming a vector field composed of a number of motion vectors between the 
first and second image, each motion vector associated to motion vector spe- 
cific parameters, 

b) forming an intermediate image by using the vector field in a way that values 
of the associated parameters of the motion vectors in a level of the interme- 
diate image are interpolated, 

c) calculating a distortion between the formed intermediate image and the 
target image, 

d) if the distortion is not acceptable, calculating new parameter values, reform- 
ing the intermediate image, and repeating step c), and 

e) if the distortion is acceptable, accepting the formed intermediate image to 
be the desired image. 

[10] An encoder according to the invention comprises at least: 



a) a vector module adapted to produce a plurality of motion vectors between a 
first and a second image, each motion vector associated with motion vector 
specific parameters, 

b) an image forming module, adapted to form an intermediate image by using 
said plurality of motion vectors to derive by interpolation values of the associ- 
ated parameters of the motion vectors at the intermediate image level, 

c) a distortion calculator module, adapted to calculate a distortion between the 
formed intermediate image and the target image, 

d) an evaluator module adapted to accept the formed intermediate image to be 
the target image if the distortion is acceptable, which is also the encoded im- 
age in the form of the motion vectors and the associated parameter values, 
and 

e) a calculator module adapted to calculate new associated parameters of the 
motion vectors, if the distortion was not acceptable, and a reforming module 
adapted to reform the intermediate image by using the new associated pa- 
rameters and to repeat the actions of module c) by transporting the reformed 
intermediate image to module c), if the distortion was not acceptable. 

[1 1] A decoder according to the invention comprises at least: 

f) a first construction module adapted to form a vector field that is composed 
of a number of motion vectors between a first and a second image, using pre- 
determined information of the motion vectors and associated parameter val- 
ues, and 

g) a second construction module adapted to form an image using the vector 
field and the associated parameter values, the formed image being the de- 
coded image. 



BRIEF DESCRIPTION OF THE DRAWINGS 



[12] In the following the invention is described in more detail by means of FIGs. 
1 - 6 in the attached drawings where: 

FIG. 1 illustrates a simplified example of a known interpolation technique, 
FIG. 2 illustrates an example of how an intermediate image is formed by using 

the inventive arrangement, 
FIG. 3 illustrates an example of different possibilities when adjusting the shape 

and color parameters for the desired image, 
FIG. 4 illustrates an example of a flow chart representing the inventive method, 
FIG. 5 shows an example of an encoder/decoder arrangement according to 

the invention, 

FIG. 6 shows an example of a motion vector in a vector field. 

DETAILED DESCRIPTION OF THE INVENTION 
[13] FIG. 2 illustrates an example of how an intermediate image is formed by 
using the inventive arrangement. The vector field, which is formed between origi- 
nal pictures IMG. 21 and IMG. 22 is that kind of a vector field, which singularly 
defines a desired interpolated picture 23. In a singular vector field, each point in 
an intermediate image has one and only one corresponding point in both original 
pictures. Thus the motion vectors in the singular vector field do not cross each 
other. This kind of vector field can be achieved, for example, in a way described 
in the inventor's other patent application WO 01/48695 A1, which is herebye in- 
corporated by reference. Since the singular vector field is described in this pub- 
lished document, the vector field or the motion vectors are not described in detail 
in this text. 

[14] The first picture IMG. 21 contains an object of a circle 24. The second pic- 
ture IMG. 22 contains an object of a triangle 25. The transformation of the object 
from the circle to the triangle represents a motion between the original pictures. 
The picture 23, which is interpolated, contains an object 26 that is an intermedi- 



ate form from the circle and the triangle. The intermediate form, which is known 
beforehand, is achieved using motion vectors, which have companion points, i.e. 
pixel values, in the circle in IMG. 21 , and in the triangle in IMG. 22. In this con- 
text, knowing the intermediate form beforehand means that the image that the in- 
termediate image is going to present exists in reality, or the intermediate image is 
a new image that the user wants to form. For example, when intermediate pic- 
tures are encoded using interpolation, or when a new image is edited from two 
pictures. Yet another example arises when a sequence of pictures exists, such 
as regular video. As sending all the intermediate images is wasteful, it is well a 
well-known method to compress the images by selecting two non-consecutive 
pictures and supplying data that will allow the receiver to reconstruct the images 
intermediate the two. Thus, the encoding, or compressing side have the full pic- 
ture at its disposal, and the receiving side will be able to reconstruct the images 
based on the interpolation or other compressed data, such as codevectors mo- 
tion vectors, field vectors, difference data, etc. Pixel P1 in the circle has a com- 
panion pixel P1B in the triangle. Pixel P1A in the intermediate object 26 is a re- 
sult of the interpolation from pixels P1 and P1B. Motion vector 27 connects these 
pixels. Pixel P2 in the circle has a companion pixel P2B in the triangle. Pixel 
P2A in the intermediate object 26 is a result of the interpolation from pixels P2 
and P2B. Motion vector 28 connects these pixels. In a similar way, all the motion 
vectors, which go through the objects, are used for the interpolation of the de- 
sired object 26. However, when interpolating the pixels of the intermediate im- 
age, the direct interpolation may not achieve the required accuracy, especially if 
the intermediate image contains nonlinear information as compared to the origi- 
nal images. Thus the values of the interpolation parameters values must be ad- 
justed to achieve the image, which is desired to be formed, i.e. the target image. 

[15] If the singular motion vector described in the inventor's PCT application WO 
01/48695 A1 is used, the motion vector L, see FIG. 6, attached to the point x 
combines the points gr(x-t(x)) (Compare P1 in Fig. 2) in the first, and g^x+t{x)) 



(Compare P1B in Fig. 2) in the second original image. It is convient to think 
about a situation where the attaching point x is on the level (timeline position) of 
the intermediate image, but the x can also be on another level between the origi- 
nal images. Here, the pixel values of the original images are gi and g 2 , and the 
movement related to the vector attached in point x is t(x), These points are used 
to form an image point g c (x+ct(x)) (Compare P1 A in Fig. 2) in the intermediate 
image: the parameter c that in general gets values from -1 to 1 describes the 
shape of the objects in the intermediate image. If c=-1, the shape information is 
the same in images g c and g 1t (i.e. in Fig. 2 object 24 equals with object 26) and 
with value c=1 the shape information of g c is the same as the one of g 2 (i.e. in 
Fig. 2 object 25 equals with object 26) 

[16] The color of the point gf c (x+ct(x)) (P1 A in Fig. 23) can be determined as a 
weighted sum of points g?(x-t(x)) (P1 in Fig. 23) and Q2(x+t(x)) (P1B in Fig. 23), 
i.e., g c (x+ct(x)) = (1-w)gi*(x-t(x)) + w g2 (x+t(x)). The parameter w that may be 
called the color parameter, o<=w<=1 , controls the source of the color informa- 
tion: with a small value of w, the color is closer to the one in image g-i, and with a 
large value w the color information is closer to the one of image g 2 . 

[17] As can be noticed, the shape and color parameters can be separately de- 
fined for each pixel of the intermediate picture. However, the values of the shape 
parameter c and the color parameter w can be preferably defined to be constant 
over the whole image, or they can be determined separately for each image 
area, e.g. an image block of 8-by-8 pixels. 

[18] After the normal interpolation for forming the intermediate picture the ad- 
justment is done by comparing the images (in a case of encoding the target im- 
age and the interpolated image) or image areas (in an extreme case comparing 
individual pixels) created by using various parameter values to the desired image 
or image area. In the first phase, the candidates for the parameter values are se- 
lected using a coarse grid, and the parameter value pair that produces the best 
intermediate image is selected. Then new images are created using the parame- 



ter values (c, w) that are close to the best ones found in the previous step, and 
the new images are also compared to the desired image. These steps are re- 
peated many times until the intermediate image is close enough to the desired 
image. This iteration can be done, for example, using a logarithm search. 

[1 9] For example, by changing the value of the shape parameters related to the 
motion vectors 28 and 27 in FIG. 2, we can create any object that has a shape 
between the shapes illustrated in the original images IMG. 21 and IMG. 22. The 
shape of the object 26 is one example of the intermediate shape of the shapes of 
objects 24 and 25. In addition, we can select the color of the intermediate image 
to get any value between the colors of objects 24 and 25. It should be noticed 
that the adjustment can also be made only for a part or parts of the intermediate 
image, for example, for the part that contains fast motion, and for example an 
area of 8 * 8 pixels. So, it is possible to divide the image into several blocks, and 
each block, i.e. part, can be formed independently from the other blocks. Due to 
this, there are a great number of variations for the image to be formed. 

[20] Furthermore, after the adjustment is done it is possible to use additional ad- 
justing parameters, which correct either the pixel values of the intermediate im- 
age or the values of the motion vectors of the singular vector field. By using this 
kind of additional adjusting parameters we can bring the resulting image closer to 
the desired one. 

[21] So, the parameter values (c, w) used with the motion vectors, the associ- 
ated parameters, contain motion and color information. When the intermediate 
picture is known beforehand, the parameter values used when creating the inter- 
polated image can be calculated to the desired values using any suitable iteration 
method, such as a logarithm search. Also the adjusting parameter values can be 
calculated using any suitable iteration method. 

[22] For example, using a method of the logarithm search, a search step of the 
parameter values is decreased during the iteration process. If the valid values of 
the shape (c) and color (w) parameters are lying between the values -1 and 1, 



the initial search step can be, for example, +-0,5 around the initial guess (shape; 
color) = (0.0, 0.0). In the first step, a total of nine images are created, with the pa- 
rameter values (0.0+-d1; 0.0+-d2) where d1 and d2 get values -0.5, 0.0, and 
+0.5, i.e., (-0.5; -0.5), (0.0; -0.5), (0.5; -0.5), (-0.5; 0.0), (0.0; 0.0), (0.5; 0.0), (-0.5; 
0.5), (0.0; 0.5), and (0.5; 0.5). The next search step may be +-0,5/2 around the 
best values found thus far, e.g. if the best value of the previous step would be 
(0.5; 0.5), the next step would include the creation of the images with parameter 
values (0.5+-d1 ; 0.5+-d2), where d1 and d2 get values -0.25, 0.0, and +0.25; 
however the combination of d1 = 0.0 and d2 = 0.0 is excluded, because it has 
been already calculated during the previous step. This is continued until the dis- 
tortion between the desired picture and the interpolated picture is within satisfac- 
tory limits or a number of predetermined iteration rounds have been performed. It 
should be noticed that there exists several different methods of the logarithm 
search. For instance, the distortion can be calculated using a direct error or a 
mean square error. 

[23] Also the adjusting parameter can be achieved in any of the ways described 
above, or any other convenient manner, the selection of which is a matter of 
technical choice to the person skilled in the art. 

[24] FIG. 3 illustrates an example of different possibilities when adjusting the as- 
sociated parameters of the motion vectors for the desired image. The first picture 
31 represents a circle, and the second picture 32 represents a triangle. The verti- 
cal axis describes values of the motion parameters, and the horizontal axis val- 
ues of the color parameters. If linear interpolation is used when forming an inter- 
mediate picture, object 34 is achieved. However in this case, the color informa- 
tion of the pixels of the intermediate picture is desired to be the same as in the 
first picture 31, i.e. zero. The motion information is desired to be the same as ob- 
ject 34 has, i.e. 0,5. Thus object 33 is the best image for representing the target 
picture. Object 33 is achieved by changing the weight of the color parameter 
value. It should be noted, that nonlinearity can be taken into account in both pa- 



rameters: one describing color and the other one describing motion, ie. shape. If 
object 33 is still not good enough, the final adjustment can be made using the ad- 
justing parameters. 

[25] FIG. 4 illustrates an example of a flow chart representing the inventive 
method. First, a singular vector field (or any other vector field) must be formed 41 
between two images. As mentioned, singular means that each motion vector in 
the vector field has points in both the images, which represent the point of the 
motion vector in an intermediate image, and the motion vectors do not cross be- 
tween the original images. 

[26] Next, an intermediate image is formed using the vector field by interpolating 
42 values of the parameter pair, i.e. values for the shape and color, in a level, i.e. 
timeline position, of the intermediate image. A distortion between the formed in- 
termediate image and the image which is desired to be formed, is calculated 43. 
A distortion can be the sum of errors of each pixel value or the sum of square er- 
rors of each pixel value, just to mention some ways. If the distortion is at an ac- 
ceptable level 44, the formed intermediate image is accepted 45. 

[27] If the distortion is not acceptable 44, new values of the parameter pair in the 
intermediate image are calculated 46. The new values can be any values be- 
tween the values in the original images. When the new parameter values have 
been calculated, the intermediate image is reformed 47. 

[28] The calculation can be done in an iterative way, using for example a loga- 
rithm search, if the calculation must be repeated several times. In each round the 
latest parameter values are used for forming the latest picture, which is com- 
pared to the previously formed picture 44. If the latest picture is better than the 
previous picture, the latest parameter values are better and they are selected to 
be initial values for the next round. In the opposite case, the previous parameter 
values are better and they remain as the initial values. When calculating the new 
parameters 46, the range of new values around the latest initial values decreases 
in each round. 
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[29] In a similar way, the additional adjusting parameters can be calculated, if 
the error is still too large after the iteration described above. 

[30] The inventive method can be used in any possible application, arrange- 
ment, and device, which needs image interpolation. For example, encoding and 
decoding video files is a very obvious environment to use the invention. In fact, 
the invention can be used in any digital image editing. 

[31] FIG. 5 shows an example of an encoder/decoder arrangement according to 
the invention. The original image file 51 is going to be encoded using the interpo- 
lation according to the invention. The image file can, for example, be a video file, 
which is composed of separate images, one after the other. The image file is in- 
serted 56 into an encoder 52, which uses the inventive interpolation. Using the 
interpolation, it is possible to decrease the size of the image file considerably. 
Let's say, for example, that every second separate image is going to be encoded 
using the inventive interpolation, and every second separate image is going to be 
retained (and if desired, encoded using another encoding technique). 

[32] The images maintained are used for creating singular vector fields between 
them. Next, the images to be encoded are interpolated. In other words, the pa- 
rameter values (and if necessary the values of the adjusting parameters in the in- 
termediate image) used with the motion vectors in the vector fields contain the 
encoding information. Iteration is used so that the images to be encoded will be 
equal with the original pictures to within an acceptable accuracy. When the en- 
coded pictures are acceptable, the encoded file 53 can be used as desired. It will 
be clear that the desirable reduction in the image file size benefits from reducing 
the number of original pictures and replacing those originals with interpolated pic- 
tures. In other words, one vector field can be used for encoding several images. 
So the whole encoded image file contains primarily encoded frames (encoded 
using a suitable encoding technique), data of the interpolated frames, and data of 
the values of the associated parameters. 
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[33] The encoded image file is sent 57 to a decoder 54, which uses the inventive 
interpolation for decoding, using the values of shape and color parameters (and if 
necessary the values of the additional adjusting parameters in the intermediate 
image). Often, the encoder and the decoder are at different places, and con- 
nected via the Internet. For example, the encoder can be in a server of a service 
provider and the decoder in a customer's terminal It is far more efficient to send 
an encoded, rather than unencoded, file. The decoder converts the encoded file 
back to the original (or almost) size using the parameters of the vector fields, and 
substantially inverting the process described above. The converted image file 55 
is delivered 58 from the decoder to a final application. 

[34] It will be clear to those skilled in the art that besides a primarily hardware 
based embodiment of the invention, the invention may also be embodied by 
software operating on a general purpose computer, a specialized computer, or a 
combination of any of the above. Thus the invention further extends to a software 
program that when executed by a computer will cause it to perform substantially 
in accordance with such aspects of the invention as the encoder of decoder de- 
scribed above, and to any media carrying such software. 

[35] Although, the invention has been described in this text using only a few ex- 
amples, it is obvious that the invention can be implemented in many different so- 
lutions. For example, error calculation can be specific either for each pixel or for 
a group of pixels (such as an 8*8 area wherein a mean error of pixels is going to 
be adjusted into an acceptable level). Thus the invention is not restricted to the 
above-mentioned examples, but the invention can used in any suitable imple- 
mentation in the scope of the inventive ideas. 
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